Skip to content

Defer loading rex/metasm/octokit/etc dependencies#21342

Merged
adfoster-r7 merged 1 commit intorapid7:masterfrom
adfoster-r7:defer-loading-dependencies
May 1, 2026
Merged

Defer loading rex/metasm/octokit/etc dependencies#21342
adfoster-r7 merged 1 commit intorapid7:masterfrom
adfoster-r7:defer-loading-dependencies

Conversation

@adfoster-r7
Copy link
Copy Markdown
Contributor

@adfoster-r7 adfoster-r7 commented Apr 21, 2026

Defer loading dependencies

Requires

Verification

  • Ensure CI passes
  • Spot check a few modules to ensure they work as expected

@adfoster-r7 adfoster-r7 changed the title Defer loading dependencies Defer loading rex/metasm/octokit/etc dependencies Apr 21, 2026
@adfoster-r7 adfoster-r7 force-pushed the defer-loading-dependencies branch from b07a040 to 9adf8e8 Compare April 21, 2026 12:14
@adfoster-r7 adfoster-r7 marked this pull request as ready for review April 21, 2026 12:14
@adfoster-r7 adfoster-r7 force-pushed the defer-loading-dependencies branch from 9adf8e8 to 0724b0d Compare April 21, 2026 12:17
@adfoster-r7 adfoster-r7 force-pushed the defer-loading-dependencies branch from 0724b0d to b4d2c10 Compare April 30, 2026 22:37
@adfoster-r7 adfoster-r7 requested a review from Copilot April 30, 2026 22:38
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR defers loading several heavy gem dependencies (dnsruby/metasm/rkelly/octokit/bcrypt/recog/zip) to reduce boot-time cost, primarily by replacing eager requires with autoloads and localized requires, and by avoiding dnsruby constants in Rex::Proto::DNS::StaticHostnames.

Changes:

  • Add global gem autoloads in msf_autoload.rb and move some requires (e.g., octokit, rkelly) to first-use sites.
  • Update Rex DNS static hostnames to use locally-defined A/AAAA type constants instead of referencing dnsruby during boot, with added spec coverage.
  • Update bundled dependencies (Gemfile.lock) and schema.rb to reflect upstream gem updates.

Reviewed changes

Copilot reviewed 19 out of 20 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
spec/lib/rex/proto/dns/static_hostnames_spec.rb Adds assertions that local DNS type constants match dnsruby types.
modules/payloads/stages/netware/shell.rb Removes eager require 'metasm' for deferred loading.
modules/payloads/stagers/netware/reverse_tcp.rb Removes eager require 'metasm' for deferred loading.
lib/rex/proto/http/response.rb Defers rkelly loading to get_html_scripts; trims whitespace.
lib/rex/proto/dns/static_hostnames.rb Introduces local TYPE_A / TYPE_AAAA to avoid dnsruby load at boot.
lib/rex/proto/dns/resolver.rb Removes eager require 'dnsruby' (now relies on deferred loading).
lib/rex/proto/dns/packet.rb Removes eager require 'dnsruby' (now relies on deferred loading).
lib/rex.rb Adds Rex-level autoloads for several subcomponents; removes eager requires for those.
lib/msf_autoload.rb Adds top-level autoloads for common gems (BCrypt/Dnsruby/Metasm/Recog/Zip).
lib/msf/util/exe/common.rb Removes eager require 'metasm'.
lib/msf/util/document_generator/pull_request_finder.rb Defers octokit require until initialization.
lib/msf/core/payload.rb Removes eager require 'metasm'.
lib/msf/core/framework.rb Removes eager require 'rex/socket/ssl' (still loaded via cert provider path).
lib/msf/core/exploit/remote/smb/client.rb Removes eager require 'recog' to rely on deferred loading.
lib/msf/core/exe/segment_injector.rb Removes eager require 'metasm'.
lib/msf/core/exe/segment_appender.rb Removes eager require 'metasm'.
lib/msf/core/db_manager/user.rb Removes eager require 'bcrypt' to rely on deferred loading.
lib/metasploit/framework.rb Removes eager requires for several gems now intended to load on demand.
db/schema.rb Updates schema version and adds new columns reflecting upstream model changes.
Gemfile.lock Bumps several gem versions required for the dependency deferral work.

Impact Analysis:

  • Blast radius: high (boot/loading path touches Rex and Msf initialization; downstream consumers include most framework entrypoints and modules); some effects are Unknown outside typical require 'msf' load order.
  • Data and contract effects: adds check_code/check_detail columns and updates schema version; dependency versions changed (potential compatibility impacts with external tooling/plugins are Unknown from diff).
  • Rollback and test focus: rollback is mostly reverting require/autoload changes plus gem lock/schema; focus validation on DNS resolution paths, payload/exe generation paths that use Metasm, and any callers using Rex DNS components outside the normal msf_autoload initialization path.

Comment thread lib/rex/proto/dns/resolver.rb
Comment thread lib/rex/proto/dns/packet.rb
@adfoster-r7 adfoster-r7 force-pushed the defer-loading-dependencies branch from b4d2c10 to 715e41e Compare April 30, 2026 23:04
@adfoster-r7 adfoster-r7 force-pushed the defer-loading-dependencies branch from 715e41e to 557ff0d Compare April 30, 2026 23:08
@cgranleese-r7 cgranleese-r7 self-assigned this May 1, 2026
@cgranleese-r7 cgranleese-r7 added enhancement rn-enhancement release notes enhancement labels May 1, 2026
@cgranleese-r7
Copy link
Copy Markdown
Contributor

Benchmarks:

require 'rex' (10 runs):

Benchmark 1: upstream-master
  Time (mean ± σ):     749.8 ms ±  18.3 ms

Benchmark 2: PR #21342
  Time (mean ± σ):     645.1 ms ±  18.7 ms

Summary
  PR #21342 ran 1.16 ± 0.04 times faster than upstream-master


msfconsole -q -x exit (3 runs):

Benchmark 1: upstream-master
  Time (mean ± σ):      2.025 s ±  0.015 s

Benchmark 2: PR #21342
  Time (mean ± σ):      1.918 s ±  0.020 s

Summary
  PR #21342 ran 1.06 ± 0.01 times faster than upstream-master

@cgranleese-r7
Copy link
Copy Markdown
Contributor

Release Notes

Defers the loading of some dependencies to improve console boot time.

@github-project-automation github-project-automation Bot moved this from Todo to In Progress in Metasploit Kanban May 1, 2026
@adfoster-r7 adfoster-r7 merged commit bbb2452 into rapid7:master May 1, 2026
63 checks passed
@adfoster-r7 adfoster-r7 deleted the defer-loading-dependencies branch May 1, 2026 13:37
@github-project-automation github-project-automation Bot moved this from In Progress to Done in Metasploit Kanban May 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement rn-enhancement release notes enhancement

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants